MATRIX: MAny-Task computing execution fabRIc at eXascale

Efficiently scheduling large number of jobs over large-scale distributed systems is critical in achieving high system utilization and throughput. Today’s state-of-the-art job management systems have predominantly Master/Slaves architectures, which have inherent limitations, such as scalability issues at extreme scales (e.g. petascales and beyond) and single point of failure. In designing the nextgeneration distributed job management system, we must address new challenges such as load balancing. This paper presents the design, analysis and implementation of a distributed execution fabric called MATRIX (MAny-Task computing execution fabRIc at eXascale). MATRIX utilizes an adaptive work stealing algorithm for distributed load balancing, and distributed hash tables for managing task metadata. MATRIX supports both high-performance computing (HPC) and many-task computing (MTC) workloads, as well as task dependencies in the execution of complex large-scale workflows. We have evaluated it using synthetic workloads up to 4K-cores on an IBM Blue Gene/P supercomputer, and have shown high efficiency rates (e.g. 85%+) are possible with certain workloads with task granularities as low as 64ms. MATRIX has shown throughput rates as high as 13K tasks/sec at 4K-core scales (one to two orders of magnitude higher than existing centralized systems). We also explore the feasibility of adaptive work stealing up to 1M-node scale through simulations.